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Paninski  laboratories  combined  their  efforts  in  this  interdisciplinary  project,  with  the  ultimate  goal  of  decoding  and  modulating  the  neuronal 
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Scientific  Progress 


Miller  et.  al.  (2015) 


Significance: 

This  study  demonstrates  that  neuronal  groups  or  ensembles,  rather  than  individual  neurons,  are  emergent  functional  units  of 
cortical  activity.  We  show  that  in  the  presence  and  absence  of  visual  stimulation,  cortical  activity  is  dominated  by  coactive 
groups  of  neurons  forming  ensembles.  These  ensembles  are  flexible  and  cannot  be  accounted  for  by  the  independent  firing 
properties  of  neurons  in  isolation.  Intrinsically  generated  ensembles  and  stimulus-evoked  ensembles  are  similar,  with  one  main 
difference:  Whereas  intrinsic  ensembles  recur  at  random  time  intervals,  visually  evoked  ensembles  are  time-locked  to  stimuli. 
We  propose  that  visual  stimuli  recruit  endogenously  generated  ensembles  to  represent  visual  attributes. 

Major  Findings: 

We  performed  two-photon  calcium  imaging  of  populations  of  neurons  from  the  primary  visual  cortex  of  awake  mice  during  visual 
stimulation  and  spontaneous  activity.  Under  both  conditions,  cortical  activity  is  dominated  by  coactive  groups  of  neurons, 
forming  ensembles  whose  activation  cannot  be  explained  by  the  independent  firing  properties  of  their  contributing  neurons, 
considered  in  isolation. 

Individual  Neurons  Contribute  Flexibly  to  Multiple  Ensembles. 

Individual  neurons  flexibly  join  multiple  ensembles,  vastly  expanding  the  encoding  potential  of  the  circuit. 

Visually  Evoked  Ensembles  Are  Similar  to  Spontaneous  Ensembles 

The  same  coactive  ensembles  can  repeat  spontaneously  and  in  response  to  visual  stimuli,  indicating  that  stimulus-evoked 
responses  arise  from  activating  these  intrinsic  building  blocks. 

Although  the  spatial  properties  of  stimulus-driven  and  spontaneous  ensembles  are  similar,  spontaneous  ensembles  are  active 
at  random  intervals,  whereas  visually  evoked  ensembles  are  time-locked  to  stimuli. 

Conclusion: 

Neuronal  ensembles,  built  by  the  co-activation  of  flexible  groups  of  neurons,  are  emergent  functional  units  of  cortical  activity 
and  propose  that  visual  stimuli  recruit  intrinsically  generated  ensembles  to  represent  visual  attributes 

Yang  et.  al.  (2015) 

Significance: 

We  are  optimistic  about  the  future  of  volumetric  imaging  in  scattering  tissue,  and  consider  SLM-based  multiregion  imaging  as 
but  one  implementation  of  a  general  strategy  of  computationally  enhanced  projective  imaging,  which  will  make  possible  the 
ability  to  interrogate  neurons  over  a  very  large  area,  with  high  temporal  resolution  and  SNR. 

Major  Findings: 

Imaging  the  functional  activity  of  large  populations  of  neurons  within  the  brain  could  be  critical  towards  understanding  the 
structure  and  function  of  neural  circuits. 

Here  we  present  a  simple  holographic  microscopy  method  to  simultaneously  image  the  neuronal  activity  of  the  mouse  cortex  in 
vivo,  across  multiple  cortical  layers  (L2/3  and  L5). 

We  demonstrate  successful  simultaneous  3D  multilayer  in  vivo  imaging  with  a  hybrid  SLM  multibeam-scanning  approach  that 
leverages  spatiotemporal  sparseness  of  activity  and  prior  structural  information  to  efficiently  extract  single  cell  neuronal  activity. 

We  can  either  extend  the  effective  area  that  can  be  sampled,  target  multiple  axial  planes  over  an  extended  range,  >  500  Dm, 
or  both,  at  depth  within  the  cortex. 

The  regional  targeting  is  performed  remotely,  through  holography,  without  any  motion  of  the  objective,  which  makes  the 
technique  a  strong  complement  to  3-D  two-photon  activation. 

The  module  allows  for  simple  and  rapid  software  based  multiplexing,  with  low  cost,  and  offers  considerable  flexibility  and 
performance  for  a  variety  of  imaging  paradigms,  and  creates  a  powerful  tool  to  study  the  functional  activity  of  neural  circuits 
across  multiple  areas  in  the  brain. 

Conclusion: 

This  imaging  approach  -  multiplexed  excitation  combined  with  computational  source  separation  and  reconstruction  -  may  be 
ideal  for  future  high  speed  in  vivo  volumetric  imaging  in  scattering  tissue. 


Carillo  Reid  et.  al.  (2015) 


Major  Findings: 

Although  the  functional  properties  of  individual  neurons  in  primary  visual  cortex  have  been  studied  intensely,  little  is  known 
about  how  neuronal  groups  could  encode  changing  visual  stimuli  using  temporal  activity  patterns. 

Identification  of  neuronal  ensembles  in  multineuronal  recordings 

We  used  in  vivo  two-photon  calcium  imaging  to  record  the  activity  of  neuronal  populations  in  primary  visual  cortex  of  awake 
mice  in  the  presence  and  absence  of  visual  stimulation.  Multidimensional  analysis  of  the  network  activity  allowed  us  to  identify 
neuronal  ensembles  defined  as  groups  of  cells  firing  in  synchrony. 

Evoked  neuronal  ensembles  recapitulate  spontaneous  ones 

These  synchronous  groups  of  neurons  were  themselves  activated  in  sequential  temporal  patterns,  which  repeated  at  much 
higher  proportions  than  chance  and  were  triggered  by  specific  visual  stimuli  such  as  natural  visual  scenes. 

Hebbian  cell  assemblies  in  cortical  microcircuits 

Sequential  patterns  were  also  present  in  recordings  of  spontaneous  activity  without  any  sensory  stimulation  and  were 
accompanied  by  precise  firing  sequences  at  the  single-cell  level.  Intrinsic  dynamics  could  be  used  to  predict  the  occurrence  of 
future  neuronal  ensembles. 

Conclusion: 

Our  data  demonstrate  that  visual  stimuli  recruit  similar  sequential  patterns  to  the  ones  observed  spontaneously,  consistent  with 
the  Hebbian  hypothesis  that  already  existing  neuronal  ensembles  firing  in  predefined  temporal  sequences  could  be  the 
microcircuit  substrate  that  encodes  visual  percepts  changing  in  time. 

Pnevmatikakis  et.  al.  (2015) 

Significance: 

Calcium  imaging  as  a  neural  recording  method  can  provide  rich  and  diverse  datasets  depending  on  many  factors  (imaging 
technique,  experimental  conditions,  calcium  indicators  etc),  and  multineuronal  ground  truth  spiking  data  is  scarce  and  typically 
hard  to  obtain.  As  such,  the  goal  of  a  globally  operating  algorithm  that  requires  no  human  intervention  and  has  performance 
guarantees  remains  elusive.  Nevertheless,  we  hope  that  the  family  of  methods  discussed  here,  under  the  unifying  umbrella  of 
constrained  matrix  factorization,  will  provide  a  useful  framework  for  future  research. 

Major  Findings: 

Developed  a  modular  approach  for  analyzing  calcium  imaging  recordings  of  large  neuronal  ensembles. 

A  flexible  and  efficient  model  for  calcium  deconvolution 

Simultaneous  identification  of  the  locations  of  the  neurons,  demix  spatially  overlapping  components,  and  denoise  and 
deconvolve  the  spiking  activity  from  the  slow  dynamics  of  the  calcium  indicator.  This  builds  upon  and  extends  the  fast  non¬ 
negative  deconvolution  (FOOPSI)  method  of  Vogelstein. 

Constrained  matrix  factorization  provides  a  flexible  framework  for  analysis  of  large-scale  calcium  imaging  data 
We  developed  a  matrix  factorization  approach  that  expresses  the  spatiotemporal  fluorescence  activity  as  the  product  of  a 
spatial  matrix  that  encodes  the  spatial  footprint  of  each  neuron  in  the  optical  field  and  a  temporal  matrix  that  characterizes  the 
calcium  concentration  of  each  neuron  overtime. 

Segmentation  of  large-scale  video  data 

Using  in  vivo  mouse  VI  spontaneous  activity  data  and  the  greedy  algorithm  which  efficiently  identifies  neurons  with  very  few 
visually-apparent  false  positives  and  denoises  the  calcium  activity  of  each  cell. 

A  modular  approach  towards  automated  calcium  imaging  data  analysis 

This  framework  is  combined  with  a  novel  constrained  deconvolution  approach  that  extracts  estimates  of  neural  activity  from 
fluorescence  traces,  to  create  a  spatiotemporal  processing  algorithm  that  requires  minimal  parameter  tuning.  A  specific 
objective  in  this  work  was  to  create  a  framework  that  requires  minimal  human  intervention 

Conclusion: 

We  demonstrate  the  general  applicability  of  our  method  by  applying  it  to  in  vitro  and  in  vivo  multi-neuronal  imaging  data,  whole 
brain  light-sheet  imaging  data,  and  dendritic  imaging  data. 

Technology  Transfer 
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Figure  1-  Mulitplane  SLM  microscope  schematic  and  data  (Yang  et.  al.  2015  submitted 
manuscript  Neuron) 

Figure  2-  NMF  Method  with  mouse  VI  data 


(4)  Statement  of  the  problem(s)  studied: 

(Yuste)  How  do  neuronal  groups  encode  changing  visual  stimuli  using  temporal  activity  patterns? 
What  new  technologies  can  we  develop  so  we  can  see  wider  and  deeper  in  mouse  brains? 

(Paninksi)  Calcium  imaging  pose  significant  challenges  to  statistical  neuroscientists,  there  are  3 
major  problems:  (i)  identifying  the  spatial  footprint  of  each  neuron  in  the  optical  field,  (ii)  demixing 
spatially  overlapping  neurons  (where  overlap  is  due  either  to  the  projection  of  a  3d  volume  onto  a 
2d  imaging  plane,  or  to  insufficient  spatial  resolution  in  3d  imaging  methods)  and  (iii) 
deconvolving  the  spiking  activity  of  each  neuron  from  the  much  slower  dynamics  of  the  calcium 
indicator. 


(5)  Summary  of  the  most  important  results  -  highlights  of  the  papers  and  manuscripts. 

Miller  et.  al.  (2015) 

Significance: 

This  study  demonstrates  that  neuronal  groups  or  ensembles,  rather  than  individual  neurons,  are 
emergent  functional  units  of  cortical  activity.  We  show  that  in  the  presence  and  absence  of  visual 
stimulation,  cortical  activity  is  dominated  by  coactive  groups  of  neurons  forming  ensembles. 

These  ensembles  are  flexible  and  cannot  be  accounted  for  by  the  independent  firing  properties  of 
neurons  in  isolation.  Intrinsically  generated  ensembles  and  stimulus-evoked  ensembles  are 
similar,  with  one  main  difference:  Whereas  intrinsic  ensembles  recur  at  random  time  intervals, 
visually  evoked  ensembles  are  time-locked  to  stimuli.  We  propose  that  visual  stimuli  recruit 
endogenously  generated  ensembles  to  represent  visual  attributes. 

Major  Findings: 

We  performed  two-photon  calcium  imaging  of  populations  of  neurons  from  the  primary  visual 
cortex  of  awake  mice  during  visual  stimulation  and  spontaneous  activity.  Under  both  conditions, 
cortical  activity  is  dominated  by  coactive  groups  of  neurons,  forming  ensembles  whose  activation 
cannot  be  explained  by  the  independent  firing  properties  of  their  contributing  neurons,  considered 
in  isolation. 

Individual  Neurons  Contribute  Flexibly  to  Multiple  Ensembles. 

Individual  neurons  flexibly  join  multiple  ensembles,  vastly  expanding  the  encoding  potential  of  the 
circuit. 

Visually  Evoked  Ensembles  Are  Similar  to  Spontaneous  Ensembles 

The  same  coactive  ensembles  can  repeat  spontaneously  and  in  response  to  visual  stimuli, 
indicating  that  stimulus-evoked  responses  arise  from  activating  these  intrinsic  building  blocks. 
Although  the  spatial  properties  of  stimulus-driven  and  spontaneous  ensembles  are  similar, 
spontaneous  ensembles  are  active  at  random  intervals,  whereas  visually  evoked  ensembles  are 
time-locked  to  stimuli. 


Conclusion: 

Neuronal  ensembles,  built  by  the  co-activation  of  flexible  groups  of  neurons,  are  emergent 
functional  units  of  cortical  activity  and  propose  that  visual  stimuli  recruit  intrinsically  generated 
ensembles  to  represent  visual  attributes 

Yang  et.  al.  (2015)  manuscript 

Significance: 

We  have  developed  volumetric  imaging  tools  for  in  vivo  imaging  in  scattering  tissue,  and  consider 
SLM-based  multiregion  imaging  as  one  implementation  of  our  general  strategy  of  computationally 
enhanced  projective  imaging,  which  will  make  possible  the  ability  to  interrogate  neurons  over  a 
very  large  area,  with  high  temporal  resolution  and  SNR. 

Major  Findings: 

Here  we  present  a  simple  holographic  microscopy  method  to  simultaneously  image  the  neuronal 
activity  of  the  mouse  cortex  in  vivo,  across  multiple  cortical  layers  (L2/3  and  L5).  (Figure  1) 

We  demonstrate  successful  simultaneous  3D  multilayer  in  vivo  imaging  with  a  hybrid  SLM 
multibeam-scanning  approach  that  leverages  spatiotemporal  sparseness  of  activity  and  prior 
structural  information  to  efficiently  extract  single  cell  neuronal  activity. 

We  can  either  extend  the  effective  area  that  can  be  sampled,  target  multiple  axial  planes  over  an 
extended  range,  >  500  mm,  or  both,  at  depth  within  the  cortex. 

The  regional  targeting  is  performed  remotely,  through  holography,  without  any  motion  of  the 
objective,  which  makes  the  technique  a  strong  complement  to  3-D  two-photon  activation. 

The  module  allows  for  simple  and  rapid  software  based  multiplexing,  with  low  cost,  and  offers 
considerable  flexibility  and  performance  for  a  variety  of  imaging  paradigms,  and  creates  a 
powerful  tool  to  study  the  functional  activity  of  neural  circuits  across  multiple  areas  in  the  brain. 

Conclusion: 

This  imaging  approach  -  multiplexed  excitation  combined  with  computational  source  separation 
and  reconstruction  -  may  be  ideal  for  future  high  speed  in  vivo  volumetric  imaging  in  scattering 
tissue. 

Carillo  Reid  et.  al.  (2015) 

Major  Findings: 

Although  the  functional  properties  of  individual  neurons  in  primary  visual  cortex  have  been 
studied  intensely,  little  is  known  about  how  neuronal  groups  could  encode  changing  visual  stimuli 
using  temporal  activity  patterns. 

Identification  of  neuronal  ensembles  in  multineuronal  recordings 

We  used  in  vivo  two-photon  calcium  imaging  to  record  the  activity  of  neuronal  populations  in 
primary  visual  cortex  of  awake  mice  in  the  presence  and  absence  of  visual  stimulation. 
Multidimensional  analysis  of  the  network  activity  allowed  us  to  identify  neuronal  ensembles 
defined  as  groups  of  cells  firing  in  synchrony. 

Evoked  neuronal  ensembles  recapitulate  spontaneous  ones 

These  synchronous  groups  of  neurons  were  themselves  activated  in  sequential  temporal 
patterns,  which  repeated  at  much  higher  proportions  than  chance  and  were  triggered  by  specific 
visual  stimuli  such  as  natural  visual  scenes. 

Hebbian  cell  assemblies  in  cortical  microcircuits 


Sequential  patterns  were  also  present  in  recordings  of  spontaneous  activity  without  any  sensory 
stimulation  and  were  accompanied  by  precise  firing  sequences  at  the  single-cell  level.  Intrinsic 
dynamics  could  be  used  to  predict  the  occurrence  of  future  neuronal  ensembles. 

Conclusion: 

Our  data  demonstrate  that  visual  stimuli  recruit  similar  sequential  patterns  to  the  ones  observed 
spontaneously,  consistent  with  the  Hebbian  hypothesis  that  already  existing  neuronal  ensembles 
firing  in  predefined  temporal  sequences  could  be  the  microcircuit  substrate  that  encodes  visual 
percepts  changing  in  time. 

Pnevmatikakis  et.  al.  (2015)  manuscript 

Significance: 

Calcium  imaging  as  a  neural  recording  method  can  provide  rich  and  diverse  datasets  depending 
on  many  factors  (imaging  technique,  experimental  conditions,  calcium  indicators  etc),  and 
multineuronal  ground  truth  spiking  data  is  scarce  and  typically  hard  to  obtain.  As  such,  the  goal  of 
a  globally  operating  algorithm  that  requires  no  human  intervention  and  has  performance 
guarantees  remains  elusive.  Nevertheless,  we  hope  that  the  family  of  methods  discussed  here, 
under  the  unifying  umbrella  of  constrained  matrix  factorization,  will  provide  a  useful  framework  for 
future  research. 

Major  Findings: 

Developed  a  modular  approach  for  analyzing  calcium  imaging  recordings  of  large  neuronal 
ensembles. 

A  flexible  and  efficient  model  for  calcium  deconvolution 

Simultaneous  identification  of  the  locations  of  the  neurons,  demix  spatially  overlapping 
components,  and  denoise  and  deconvolve  the  spiking  activity  from  the  slow  dynamics  of  the 
calcium  indicator.  This  builds  upon  and  extends  the  fast  non-negative  deconvolution  (FOOPSI) 
method  of  Vogelstein. 

Constrained  matrix  factorization  provides  a  flexible  framework  for  analysis  of  large-scale 
calcium  imaging  data 

We  developed  a  matrix  factorization  approach  that  expresses  the  spatiotemporal  fluorescence 
activity  as  the  product  of  a  spatial  matrix  that  encodes  the  spatial  footprint  of  each  neuron  in  the 
optical  field  and  a  temporal  matrix  that  characterizes  the  calcium  concentration  of  each  neuron 
over  time. 

Segmentation  of  large-scale  video  data 

Using  in  vivo  mouse  VI  spontaneous  activity  data  and  the  greedy  algorithm,  which  efficiently 
identifies  neurons  with  very  few  visually-apparent  false  positives  and  denoises  the  calcium  activity 
of  each  cell. 

A  modular  approach  towards  automated  calcium  imaging  data  analysis 

This  framework  is  combined  with  a  novel  constrained  deconvolution  approach  that  extracts 
estimates  of  neural  activity  from  fluorescence  traces,  to  create  a  spatiotemporal  processing 
algorithm  that  requires  minimal  parameter  tuning.  A  specific  objective  in  this  work  was  to  create  a 
framework  that  requires  minimal  human  intervention 

Conclusion: 

We  demonstrate  the  general  applicability  of  our  method  by  applying  it  to  in  vitro  and  in  vivo  multi¬ 
neuronal  imaging  data,  whole  brain  light-sheet  imaging  data,  and  dendritic  imaging  data. 
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Figure  1 :  A)  Schematic  of  multi-plane  microscope.  B)  Cartoon,  and  actual  data  set  of  simultaneous  dual 
plane  imaging,  here  L2/3  and  L5  of  mouse  visual  cortex,  taken  at  10  Hz.  C)  Extracted  activity  traces,  color 
coded  for  each  plane.  Note  extremely  high  SNR  D)  Interlaminar  correlation  map  between  L2/3  and  L5 
neurons.  E)  Demonstration  of  NMF  source  separation  vs  human  chosen  non-overlapped  ROIs  and  ICA 
extracted  signals.  NMF  performs  better  than  both.  (Adapted  from  Yang  et.  al.  2015  submitted  Neuron) 
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Figure  2.  Application  to  mouse  VI  in  vivo  data.  A:  Inferred  spatial  footprints  superimposed  on  the  correlation 
image  of  the  raw  data.  The  components  are  sorted  in  decreasing  order  based  on  the  maximum  temporal  value. 
B:  Extracted  (normalized)  fluorescence  traces  from  the  corresponding  spatial  components.  C-G:  Depiction  of  the 
merging  operation:  C-E:  Three  overlapping  spatial  components  with  highly  correlated  temporal  components.  F: 
Merged  spatial  component  (component  1  is  panels  A-B).  G:  Estimated  temporal  components  of  individual 
(dashed)  and  merged  (solid)  components.  The  constrained  NMF  method  can  efficiently  identify  multiple  neurons 
with  possibly  overlapping  spatial  footprints.  See  Supplementary  video  SI  for  a  clearer  spatiotemporal  view  of 
these  results.  (From  Pnevmatikakis  et.  al.  2015  submitted  Neuron) 


